Quantum melting of magnetic order due to orbital fluctuations 
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We have studied the phase diagram and excitations of the spin-orbital model derived for a three 
dimensional perovskite lattice, as in KCUF3. The results demonstrate that the orbital degeneracy 
drastically increases quantum fluctuations and suppresses the classical long-range order near the 
multicritical point in the mean-field phase diagram. This generates a qualitatively new spin liquid, 
providing the first example of a valence bond ground state in three dimensions. 

PACS numbers: 71.27.+a, 75.10.-b, 75.30.Et, 74.72.-h. 



It is common wisdom that macroscopic ensembles of 
interacting particles tend to behave classically. This is 
not always true, however, and the study of collective 
quantum systems starts to become a prominent theme 
in condensed matter physics. Central to this pursuit are 
low-dimensional quantum spin systems (spin chains |l]] 
and ladders ||), and it proves difficult to achieve quan- 
tum melting of magnetic long-range order (LRO) in em- 
pirically relevant systems in higher dimensions. Here we 
want to suggest a class of systems in which quantum- 
melting occurs even in three dimensions: small spin, or- 
bital degenerate magnetic insulators, the so-called Kugel- 
Khomskii (KK) systems ||. There might exist already 
a physical realization of such a three-dimensional (3D) 
"quantum spin-orbital liquid": LiNiC>2. 

Global SU(2) by itself is not symmetric enough to de- 
feat classical order in D > 1 and the pursuit has been 
open for some time to engineer more fluctuations into 
these systems. Three (related) strategies to realize quan- 
tum melting have proven to be successful: (i) Adding 
zero-dimensional fluctuations as in the bilayer Heisen- 
berg model which leads to an incompressible spin liquid 
||,[| . (ii) Frustrating the system so that the classical sec- 
tor gets highly degenerate, as in the case of the 5=1/2 
square lattice with longer ranged antiferromagnetic (AF) 
interactions (J1-J2-J3 models HQ). These systems in- 
volve fine-tuning of parameters and are therefore hard 
to realize by chemistry, (iii) Finally, reducing the num- 
ber of magnetic bonds, as in the 1/5 depleted square 
lattice, where the resulting plaquette resonating valence 
bond (PRVB) state explains the spin gap observed in 
CaV^g ||. In this Letter we show that orbital degen- 
eracy operates through the same basic mechanisms to 
produce quantum melting in the KK systems. The nov- 
elty is that these systems tend to " self-tune" to (critical) 
points of high classical degeneneracy. There are interac- 



tions which may lift the classical degeneracy, but they 
are usually weak. 

An interaction of this kind is the electron-phonon cou- 
pling - the degeneracy is lifted by a change in crystal 
structure, the conventional collective Jahn- Teller (JT) 
instability. However, as was pointed out in the semi- 
nal work by Kugel and Khomskii ||, in orbital degener- 
ate Mott-Hubbard insulators one has to consider in first 
instance the purely electronic problem. Because of the 
large local Coulomb interactions (Hubbard U), a low en- 
ergy Hilbert space splits off, spanned by spin and orbital 
configuration space, with superexchange-likc couplings 
between both spin and orbital local degrees of freedom. 
The orbital sector carries a discrete symmetry and the net 
outcome is that the clock-like orbital degrees of freedom 
get coupled into the SU(2) spin problem. Such a system 
might undergo symmetry breaking in states with simul- 
taneous spin- and orbital order. The lattice has to react 
to the symmetry lowering in the orbital sector, but it was 
recently convincingly shown, at least in the archetypical 
compound KCUF3, that the structural distortion is a side 
effect |J. 

The fundamental question arises if these forms of clas- 
sical order are always stable against quantum fluctu- 
ations. Although the subject is much more general 
(singlet-triplet models in rare earth compounds pfj[ , 
V 2 3 0' LaMn C>3 @, heavy fermions @), to fo- 
cus here on the simplest situation encountered in KCUF3 
and related systems ||. These are JT-distorted cubic, 3D 
analogues of the cuprate superconductors pj . The mag- 
netic ion is in a 3d 9 state, characterized in the absence 
of JT-distortion by two degenerate e g (x 2 — y 2 ~ \x), 
3z 2 — 1 ~ \z)) orbitals, next to the S — 1/2 spin degen- 
eracy. Kugel and Khomskii derived the effective Hamil- 
tonian || with AF superexchange J = t 2 /U (where t is 
the hopping between \z) orbitals along the c-axis), 
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neglecting the Hund's rule splittings of the intermediate 
d 8 states. Including those up to order r\ = Jh/U (Jh is 
the singlet-triplet splitting) yields in addition, 



(ij), 



(2) 



In Eqs. (0) and (|2|), refers to the spin at site i, while a 
refers to the cubic (a, b, c) axes, and the orbital degrees 
of freedom are represented by 



a(b) 



h-Tf ±V3rf) 



(3) 



The r's are Pauli matrices acting on the orbital pseudo- 
spins \x) = ( J J, \z) = ( ° J. Hence, we find a Heisen- 

berg Hamiltonian for the spins, coupled into an orbital 
problem which is clock-model-like (there are three direc- 
tional orbitals: 3x 2 — 1, 3y 2 — 1, and 3z 2 — 1, but they are 
not independent) [ p!5| . As we will see in a moment, the 
Hund's rule coupling term (||) acts to lift the degeneracy. 
Next to this, we introduce another control parameter, 
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(4) 



a "magnetic field" for the orbital pseudo-spins, loosely 
associated with an uniaxial pressure along the c-axis. 

The classical phase-diagram of the spin-orbital model 
H = Hi + H2 + H3, shown in Fig. 1, demonstrates 
the competition between the spin and orbital interac- 
tions similar to that found before in two dimensions 
|i6|| . It consists of five phases with staggered (two- 
sublattice) magnetic long-range order (LRO): (i) At large 
positive E z , the orbital system is uniformly polarized 
along x 2 — y 2 . As no superexchange is possible in the 
c-direction, the (a, b) planes decouple magnetically, and 
we recognize the two-dimensional (2D) antiferromagnet, 
called further AFxx, well known from the cuprate super- 
conductors, (ii) At large negative E z , the orbitals polar- 
ize along 3z 2 — 1, and the spin system is an anisotropic 
3D antiferromagnet, called AFzz. These two AF phases 
(AFxx and AFzz) are stabilized by the anisotropic su- 
perexchange which amounts to 4J between z orbitals 
along the c-axis, and to 9J/4 (J/4) between the x (z) 
orbitals in the (a, 6)-planes, respectively. In contrast, 
the Hund's rule Jh stabilizes mixed-orbital (MO) phases 
in which both AF and ferromagnetic (FM) couplings 
occur: (iii) At large Jh/U and E z < 0, a MOffa 



phase is found, characterized at each site by orbitals, 
\ia) = cos 9i\xa) + sin^lzcr), with the sign of 9i alter- 
nating between the two sublattices in the (a, b) planes. 
At small \E Z \ the orbitals stagger like x 2 — z 2 , y 2 — z 2 , 
x 2 — z 2 , ■ ■ •, and point towards each other along the c-axis 
which results there in strong (~ J) AF interactions. In 
contrast, weak FM interactions (~ Jh) occur within the 
(a, b) planes, (iv) A similar MO state, called MOaff, is 
found at large Jh/U and E z > 0, with the directional or- 
bitals staggered in FM (6, c) planes, and AF order along 
the a direction [ffTj. Both states are degenerate at the 
line E z = 0, where cos 20 j = — (1 — ^)/(2+ 3r;), and one 
is close to the KK phase H in KCuF 3 , where, within the 
limitations of chemistry, the magnetic order appears to 
be equivalent to an ideal one-dimensional (ID) 5=1/2 
spin system, (v) Finally, the MOaaf phase with a small 
|z)-amplitude, AF order in the (a, b) planes and FM order 
along the c-axis, is stable between the MOaff and AFxx 
phases. Thus, we find the same ingredient as in the frus- 
trated Heisenberg antiferromagnet (HAF): critical lines 
where different classical spin structures become degener- 
ate. The present case appears to be more extreme as even 
the effective dimensionality of the spin system changes 
because of the coupling to the orbital sector. The frus- 
tration manifests itself at the classical degeneracy point 
M = [E z ,rj) = (0,0), where the orbitals may be rotated 
freely when the spins are AF, and the spins may be ro- 
tated freely within the FM planes of the MO phases. The 
same energy of — 3J per site is obtained either in a 3D 
antiferromagnet with completely frustrated orbitals [con- 
sider (SiSi+s) = —1/4 in Eq. d3)], or in a disordered spin 
system due to the orbital sector. 

It is instructive to consider the stability of the clas- 
sical phase diagram to Gaussian quantum fluctuations 
fujfl . The collective modes can be calculated using, e.g., 
a random phase approximation (RPA) within the Green 
function technique 01| . Next to the Goldstone modes of 
the spin system, one finds optical modes corresponding 
to orbital excitations which occur both in the presence 
('transverse') and in the absence ('longitudinal') of a si- 
multaneous spin-flip. The new feature is that the spin 
and transverse orbital excitations are coupled, so that 
fluctuations in the orbital sector also affect the spin sec- 
tor. The approach of the critical lines is signalled by 
the softening of both longitudinal and transverse orbital 
modes. The (mixed) transverse modes give the domi- 
nating contribution to the renormalization of energy and 
magnetic order parameter. In the AFxx (AFzz) phase 
the lowest transverse mode softens along k = (jr,0,k z ) 
[k = (k x , 0, 0)], and equivalent lines in the Brillouin zone 
(BZ), regardless how one approaches the critical lines. 
Thus, these modes become dispersionless along partic- 
ular (soft-mode) lines in the BZ, where we find finite 
masses in the perpendicular directions, 

^AFxx(fc) -> A K + S X (hi + Uk 2 x k 2 y + fc^) 1/2 , 
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WAFzz(fc) - A z +B z (fc 2 + 4fc 2 ) , ( 5 ) 

with Aj = and Bi ^ at the M point, and the 
quantum fluctuations diverge logarithmically, (SS) ~ 
Jd 3 k/uj{k) ~ Jd 2 k/(Ai + B, t k 2 ) ~ lnAi, if A, -> 
at the transition. An analytic expansion could not be 
performed in the MO phases, but the numerical results 
reported below suggest a qualitatively similar behavior. 

We have verified that the above behavior of the soft 
mode results in large quantum corrections (SS Z ) to the 
order parameter in all magnetic phases close to the criti- 
cal lines. As an example we show (S z ) in AFxx and AFzz 
phases, being significantly lower than in a 2D HAF in a 
broad parameter regime (Fig. 2). Similarly as in two di- 
mensions |fl6| , the LRO is overwhelmed by quantum fluc- 
tuations at particular lines, where |(5S' 2: )| = (S z ) (Fig. 
1). Unlike (SS Z ), the RPA energies of the ordered phases 
show no divergence, with quite similar energy gains in AF 
and MO phases of the order of 0.6 J jljj. We therefore 
believe that RPA is here as accurate as in the pure-spin 
HAF and conclude that the classical order is destroyed 
by quantum fluctuations in the small \E Z \ and small r\ 
region between dashed and dotted lines in Fig. 1. 

Although the above theory is known to perform quite 
well in the simplest systems p0| , it might be misleading 
in more complicated situations. For instance, the "finite 
mass mode softening" occurs also in the J1-J2-J3 model 
where it is shown to be inconsequential in the large S 
limit because of an "order out of disorder" phenomenon 
. In contrast, low order spin- wave theory is blind for 
the quantum transition occurring in the bilayer model || . 
In all these cases, including ours, the quantum melting 
is promoted by the drastic enhancement of local fluctu- 
ations. It is then instructive to consider valence bond 
(VB) states with the individual spins paired into 
singlets and the orbitals optimized variationally. As the 
energy of a singlet is lowest when the orbitals point along 
the bond, the optimal states with all singlets lined up in 
parallel (see Fig. 3) are: (i) for E z > 0, singlets along 
the a-axis with orbitals close to 3a; 2 — 1 (VBa), and (ii) 
for E z < 0, singlets along the c-axis with orbitals ~ \z) 
(VBc). Both optimize spin and orbital energy on every 
second bond, and have lower energy than the classical 
states close to the classical degeneracy [M . 

Further, we included the leading quantum fluctua- 
tions in the VB states. The energy of the resonat- 
ing VBc (RVBc) state was obtained using the Bethe 
ansatz result for the ID HAF. We attempted to im- 
prove the VBa state by constructing the PRVB states, 
— (|* ) + e^|<3> 6 )), from the singlet pairs along a 
and b, \^ a ) and l^b). Surprisingly, more energy is gained 
instead in the plaquette VB (PVB) states in which the 
wave functions \^f a ) and \^b) alternate and form a super- 
lattice. The exceptional stability of these (nonresonat- 
ing) PVB states is due to a unique mechanism involving 
the orbital sector. Unlike in the HAF, the bonds not oc- 



cupied by the singlets contribute orbital energy and this is 
optimized when singlets in orthogonal directions are con- 
nected K V < 0-30, this PVB alternating (PVBA) 
state (Fig. 3) is stable at E z > 0, while a similar PVB in- 
terlayered state with alternating layers of (a, 6)-plane/c- 
axis bonds (PVBIc), and the RVBc state are stable at 
E z < 0. Finally, a state interlayered along a, but in the 
same pattern as in PVBIc, occurs in between the PVBA 
and PVBIc states. Thus, a spin liquid is stabilized by the 
orbital degeneracy over the MO phases with RPA fluctu- 
ations in a broad regime (Fig. 4). This resembles the 
situation in a 2D 1/5-depleted lattice ||, but the present 
instability is much stronger and happens in three dimen- 
sions. 

Summarizing, we find strong theoretical arguments 
supporting the conjecture that quantum-melting might 
occur in orbital degenerate Mott-Hubbard insulators. 
Why does it not occur always (e.g., in KCUF3)? Next to 
the Hund's rule coupling, the electron-phonon coupling A 
is dangerous. The lattice will react to the orbital fluctua- 
tions, dressing them up in analogy with polaron physics, 
and thereby reducing the coupling constant. In order 
to quantum melt KCuF3-like states, one should there- 
fore look for ways to reduce both the effective Jh and A. 
We believe that this situation is encountered in LiNi02: 
although the spin-spin interactions in the (111) planes 
should be very weakly FM according to the Goodenough- 
Kanamori rules, magnetic LRO is absent and the sys- 
tem might represent the spin-orbital liquid. More strik- 
ingly, LiNi02 is cubic and should undergo a collective 
JT transition, which absence is actually an old chemistry 
mystery! Upon electron-hole transformation, d 7 low-spin 
Ni 3+ maps on d 9 Cu 2+ in KCUF3, but with a difference 
in chemistry. While the e g hole in KCUF3 is nearly en- 
tirely localized on the Cu, the e g electron in LiNi02 is 
rather strongly delocalized over the Ni and surrounding 
O ions which reduces both Jh and A, and explains the 
absence of classical ordering. A more precise experimen- 
tal characterization of LiNi02 is needed. 
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FIG. 1. Mean- field phase diagram of the spin-orbital model 
(1-4) in the (E z , Jh) plane. Full lines indicate transitions be- 
tween the classical states, while LRO is destroyed above the 
dashed (below the dotted) lines for the AF (MO) phases. 

FIG. 2. Renormalization of AF LRO {St) in AFzz (left) 
and AFxx (right) phases as functions of E z /J. 

FIG. 3. Schematic representation of spin singlets (double 
lines) in the disordered states: VBa, VBc, and PVBA. In the 
PVBA state the plaquettes with singlets || a and || b alter- 
nate both in the (a, 6) planes and in the c direction, while 
the PVBIc state is obtained by interleaving successive PVBA 
(a, b) planes with one VBc double layer. 

FIG. 4. The same as in Fig. 1, but including quantum fluc- 
tuations. The spin liquid (RVBc, PVBIc, PVBIa, and PVBA) 
takes over in the shaded crossover regime. 
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